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ABSTRACT 

The  relevance  feedback  track  in  TREC  2009  focuses  on 
two  sub  tasks:  actively  selecting  good  documents  for  users  to 
provide  relevance  feedback  and  retrieving  documents  based 
on  user  relevance  feedback.  For  the  first  task,  we  tried  a 
clustering  based  method  and  the  Transductive  Experimen¬ 
tal  Design  (TED)  method  proposed  by  Yu  et  al.  [5].  For 
clustering  based  method,  we  use  the  K-means  algorithm  to 
cluster  the  top  retrieved  documents  and  choose  the  most 
representative  document  of  each  cluster.  The  TED  method 
aims  to  find  documents  that  are  hard-to-predict  and  repre¬ 
sentative  of  the  unlabeled  documents.  For  the  second  task, 
we  did  query  expansion  based  on  a  relevance  model  learned 
on  the  relevant  documents. 


1.  INTRODUCTION 

We  assume  good  documents  for  relevance  feedback  should 
firstly  be  relevant,  or  near  relevant,  since  prior  research  has 
shown  that  relevant  documents  are  more  informative  than 
non-relevant  ones  for  commonly  used  relevance  feedback  al¬ 
gorithms.  Thus  a  simple  idea  of  selecting  documents  for 
relevance  feedback  is  just  to  choose  the  top  retrieved  doc¬ 
uments.  However,  this  approach  may  return  similar  docu¬ 
ments  and  thus  suffers  from  two  problems:  diminishing  of 
return  and  all  eggs  in  one  basket.  Let’s  consider  one  case 
where  the  top  5  documents  are  very  similar  to  each  other  (or 
near  duplicates).  Knowing  all  of  them  are  relevant  won’t 
be  more  valuable  than  knowing  just  one  of  them  is  rele¬ 
vant,  which  is  the  so-called  “diminishing  of  return”  problem. 
Meanwhile,  if  one  of  them  is  non-relevant,  usually  all  of  them 
are  non-relevant,  which  is  the  so-called  “all  eggs  in  one  bas¬ 
ket”  problem.  To  avoid  these  problems,  we  hope  to  diversify 
the  selected  document  set  so  that  we  have  more  chances  of 
finding  relevant  documents  and  learning  more  relevant  fac¬ 
tors  or  terms  for  query  expansion.  Motivated  by  the  above 
intuitions,  we  tried  two  approaches  to  find  good  documents 
for  relevance  feedback  in  our  experiments.  One  approach 
is  based  on  document  clustering,  and  the  other  approach  is 
based  on  Transductive  Experimental  Design  [5]. 

For  the  retrieval  task,  we  adapt  a  language  modeling  ap¬ 
proach  based  on  the  Indri  (part  of  lemur)  information  re¬ 
trieval  toolkit.  The  language  modeling  approaches  are  widely 
used  on  several  standard  information  retrieval  benchmark 
data  sets,  and  we  want  to  see  how  they  perform  on  the  new 
web  data  ClueWeb09.  In  our  experiments,  we  go  beyond 


the  commonly  used  “bag-of-words”  models  by  incorporating 
phrase  and  text  window  match  in  our  retrieval  model. 


2.  CHOOSING  DOCUMENTS  FOR  RELE¬ 
VANCE  FEEDBACK  BASED  ON  DOCU¬ 
MENT  CLUSTERING 

There  are  two  motivations  for  us  to  choose  the  clustering 
approach.  First,  two  similar  documents  sharing  most  of  the 
common  words  won’t  be  more  informative  than  only  one 
of  them,  and  a  diverse  set  of  relevant  documents  can  help 
us  to  find  a  more  diverse  set  of  relevant  terms/factors  for 
query  expansion.  Second,  for  an  ambiguous  query,  a  group 
of  documents  representing  different  possible  user  intentions 
will  increase  the  probability  that  at  least  one  of  them  is 
relevant. 

We  use  the  K-means  algorithm  to  cluster  the  top  50  re¬ 
trieved  documents  into  5  clusters.  A  document  from  each 
cluster  C  is  chosen  as  follows: 

dc  =  argmax  similarity(di,  dj)  (1) 

1  dj  GC 


Each  document  is  represented  as  a  vector,  where  each  di¬ 
mension  is  the  TFIDF  score  of  the  corresponding  term  to: 
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where  tfm,d  is  the  term  frequency  of  term  m  in  document  d, 

Ld  is  the  length  of  d,  Lavg  is  the  average  document  length, 

N  is  the  total  number  of  documents  in  the  corpus,  dfm  is  the 
document  frequency  of  term  m,  and  k\ ,  b  are  two  parameters 
that  need  to  be  set  manually.  Cosine  similarity  is  used  to 
measure  the  distance  between  two  documents: 

similarity (dt,  dj)  =  ^  (3) 
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3.  CHOOSING  DOCUMENTS  FOR  RELE¬ 
VANCE  FEEDBACK  BASED  ON  TRANS¬ 
DUCTIVE  EXPERIMENTAL  DESIGN  (TED) 

Transductive  Experimental  Design  [5]  aims  to  select  in¬ 
stances  that  are  hard-to-predict  and  representative  of  unla¬ 
beled  instances.  More  specifically,  TED  tries  to  solve  the 
following  optimization  problem: 


min  V  \ 
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*The  work  is  done  while  the  author  is  a  visiting  student  at 
UCSC. 


subject  to  |  A|  =  K,  A  e  C 


(4) 
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Algorithm  1  Sequential  Algorithm 

Input:  Xc,  Xp,  fj,  >  0,  K 

1:  Initialization:  otij  <—  Xi  ■  ®j/||a5i||  ||a:j||  ,i  £  P ,j  £  C 
2:  repeat: 

3:  j  <r-  arg  maxieC  Eiep  ah/(ai,j  +  M) 

4:  A<-AU{j} 

5:  for  ieP,i'eC  otiy  «—  cqy  —  at,  jap j /(ajj  +  g) 
6:  until  |A|  =  K 
7:  return  A 


where  P  is  the  set  of  all  instances,  C  is  the  set  of  candidate 
instances  (i.e.  documents),  A  is  the  set  of  selected  instances. 
Xp,  Xc  and  Xa  are  the  the  matrix  representations  of  all 
instances  in  P,  C,  A,  and  K  is  the  number  of  instances  we 
want  to  select. 

TED  aims  to  find  the  optimal  set  of  examples  A  to  ap¬ 
proximate  each  instance  Xi  of  Xp.  The  approximation  can 
be  seen  as  the  regularized  projection  of  n  onto  the  linear 
subspace  spanned  by  Xa.  Therefore,  TED  has  a  geometric 
interpretation  that  it  tends  to  find  the  representative  data 
set  Xa  that  span  a  linear  subspace  that  retains  most  of  the 
information  of  the  whole  set  of  instances  Xp.  Therefore, 
given  a  sufficiently  large  set  Xp,  TED  actually  explores  the 
information  about  the  distribution  of  unlabeled  data. 

The  optimization  problem  is  NP-hard,  and  we  used  the 
sequential  algorithm  introduced  in  [6]  to  find  a  suboptimal 
solution  (Algorithm  1). 

4.  RETRIEVAL  MODELS 

In  the  second  phase,  we  tried  a  learning  to  rank  approach 
(Multiple  Addictive  Regression  Tree)  and  language  models. 
We  use  .GOV  data  set  to  tune  the  parameters  and  evaluate 
different  approaches.  Because  the  settings  of  the  TREC  ex¬ 
periments  are  very  different  from  the  standard  learning  to 
rank  scenario,  MART  does  not  perform  well  on  the  .GOV 
data  set,  thus  we  didn’t  submit  the  corresponding  results. 
Now  we  briefly  discuss  the  language  modeling  approach  used 
for  generating  the  submitted  run. 

The  language  modeling  approach  has  been  successfully 
used  in  standard  non  web  retrieval  tasks.  We  are  interested 
to  see  whether  it  can  do  well  on  the  relevance  feedback  task 
with  web  data.  Indri  [1]  (in  Lemur  toolkit)  is  a  standard 
open  source  search  engine  based  on  language  models.  It  is 
designed  to  handle  large  datasets,  and  supports  a  very  flex¬ 
ible  query  language.  Given  the  limited  time,  we  used  Indri 
as  our  primary  search  tool  throughout  our  experiments  and 
implemented  our  retrieval  models  based  on  the  Indri  query 
language. 

4.1  Baseline  Model 

In  the  traditional  “bag-of-words”  models,  the  word  posi¬ 
tion  information  is  ignored.  However,  word  position  infor¬ 
mation  might  be  of  great  value  in  some  cases.  For  example, 
for  query  “the  music  man”,  which  is  a  musical  name,  what 
the  user  really  wants  is  information  about  the  musical.  Thus 
a  relevant  document  must  contain  exactly  the  phrase  “the 
music  man”.  Besides,  intuitively  all  query  words  are  ex¬ 
pected  to  occur  closely  in  relevant  documents,  so  that  their 
combination  makes  the  document  relevant.  These  motivate 
us  to  use  both  phrase  match  and  text  window  match  in  our 
retrieval  models. 

Our  retrieval  models  are  based  on  the  multiple  Bernoulli 


Figure  1:  Baseline  retrieval  method 


model  [3]  (model  B).  In  this  model,  a  binary  random  vari¬ 
able  is  defined  for  each  feature  (which  could  be  a  word,  a 
phrase,  or  a  text  window)  to  indicate  whether  this  feature 
fi  is  present(/i  =  1)  or  absent(/;  =  0)  in  a  document  d.  We 
treat  each  position  of  this  document  as  a  sample  from  the 
model  where  some  features  have  shown  up  and  other  fea¬ 
tures  are  absent.  For  example,  in  a  document  with  content 
“A  B  C”,  the  features  present  in  the  first  position  include  the 
word  “A”,  the  phrase  “AB”  ( phr(AB )),  the  phrase  “ABA” 
(phr( ABA)),  the  text  window  of  length  2  and  containing 
both  “A”  and  “B”  (mnd2(AB)),  the  text  window  of  length 
3  and  containing  both  “A”  and  “C”  (unnd3(AC)),  and  the 
text  window  of  length  3  and  containing  “A”,  “B”,  and  “C” 
(wmd3(ABC)).  The  MLE  of  p(fi  =  l|d)  is  calculated  based 
on  the  frequency  of  feature  fi  occurring  in  d. 

In  many  cases,  the  title  field  of  a  web  page  includes  very 
important  information  about  this  page.  Prior  research  also 
found  that  the  anchor  texts  of  web  pages  are  useful  for  re¬ 
trieval,  possibly  because  they  are  usually  not  written  by  the 
page  author  and  thus  unlikely  to  be  biased.  To  take  advan¬ 
tages  of  the  prior  findings,  we  tried  a  mixture  model  with 
different  document  representations  in  our  baseline  retrieval. 
The  fields  we  used  are:  (t)itle,  (o)nchor,  (/Reading  (text  with 
hi  and  h2  tags),  and  the  whole  (d)ocument  (which  includes 
all  the  other  fields).  The  mixture  model  is: 


p(fi  =  m  = 


i 


=  l\0t)  +  \aP(fi  =  1|  0a) 


At  +  Aa  +  Ah  +  A  d 
+  A  hP{fi  =  1|  9h)  +  A  dP(fi  =  l|0d)} 


(5) 


where  fi  could  be  a  word,  a  phrase,  or  a  text  window. 

Figure  1  shows  the  baseline  retrieval  methods  with  a  query 
example  “wi  wj”.  We  assign  particular  weights  to  the  word 
match  (Ai  for  wi  and  A2  for  w 2,  which  are  chosen  to  be  equal 
in  our  experiment),  phrase  match  (A3),  and  text  window 
match  (A4)  respectively,  then  documents  could  be  ranked 
according  to: 

S0(q,d)  =Ai+^(|_Aj  +  Xt(Ai  *  logP(w\  =  l|d) 

+  A2  *  logP(w 2  =  l|d)  (6) 

+  A3  *  logP(phr(w\W2 )  =  l|d) 

+  A4  *  logP(wind8{w\W2)  =  l|d)) 


4.2  Relevance  Model 

A  relevance  model  is  estimated  based  on  the  relevant  doc¬ 
uments. 


11DV  P(w  =  l,R)  J2deu+P(w  =  l\d)P(R\d)P(d) 
P(w  =  1|  R)  =  p(R)  = - 

(7) 

D+  is  the  set  of  relevant  documents  of  the  query,  P(d)  is 
assumed  to  be  uniform  over  all  documents,  P(R\d)  is  the 
probability  of  relevance  of  document  d,  which  is  given.  Ac¬ 
cording  to  this  model,  we  select  the  top  N  words  with  largest 
probabilities  P(w  =  l|f?)  for  query  expansion. 

The  relevance  model  is  shown  in  figure  2.  The  weight  of 
each  new  word  is  set  according  to  P(w  =  1 1 7?) .  Based  on 
the  relevance  model,  we  calculate: 


Si  ( q,d ) 


Euj6Wr  p(wi  =  1| R)logP(wi  =  l|d) 
Wr  P(Wi  ~  1|7?) 


where  Wr  is  the  set  of  words  that  are  selected  for  query 
expansion. 

Combining  the  original  query  model  with  the  relevance 
model,  we  rank  the  documents  based  on: 


S(qexP,  d)  =  (1  -  Xfb)So(q,d)  +  X fbSi(q,d)  (8) 

where  Xfb  is  the  combination  weight  that  reflects  how  much 
we  believe  the  relevance  model. 


5.  EXPERIMENTAL  SETUP 

The  large  web  collection  Clueweb09  is  used  this  year.  It 
contains  around  1  billion  web  pages.  We  chose  to  work  on 
its  subset  B,  which  contains  about  fifty  million  documents. 
We  index  the  documents  using  the  Indri  toolkit  [2].  A  stop 
word  list  is  used  when  creating  the  index.  We  also  chose  to 
create  an  index  on  different  document  fields,  including  title, 
heading,  anchor,  and  the  full  document  (all  the  other  fields). 

Indri  Query  Language  that  considers  phrase  match  and 
text  window  match  is  used  in  our  experiment.  [4].  Take  the 
query  “obama  family  tree”  as  an  example,  the  indri  query 
string  would  be: 

#weight( 

Ai  obama 
Ai  family 
Ai  tree 

A2  #1  (obama  family) 

A2  #1  (family  tree) 

A2  #1  (obama  family  tree) 

A3  #MU>8(obama  family) 

A3  ^MU>8(obama  tree) 

A3  #Mw8(family  tree) 

A3  ^Mwl2(obama  family  tree) 

) 


case,  the  indri  query  string  is: 

#wsum( 

At  obama.  (title)  Xa  obama.  (anchor) 

Xh  obama  .(heading)  Xd  obama  .{document) 

) 

In  the  clustering  experiment,  we  used  K-means  to  cluster 
the  top  50  retrieved  documents  into  5  clusters.  The  reason 
why  we  chose  a  small  number  (50  in  our  case)  as  the  cut 
line  is  that  we  hope  the  chosen  documents  have  high  prob¬ 
abilities  of  being  relevant.  Equation  (2)  is  used  to  calculate 
the  document  vectors.  Cosine  similarity  measure  is  used 
to  calculate  the  distance.  When  calculating  the  document 
vectors,  and  b  are  set  as  1.2  and  0.75  respectively. 

In  the  TED  experiments,  we  kept  the  top  1000  retrieved 
documents,  which  consist  of  the  set  P  (see  section  3).  The 
candidate  set  C  is  the  top  50  documents.  The  sequential  al¬ 
gorithm  introduced  in  section  3  is  used  to  select  5  best  doc¬ 
uments  from  the  top  50  based  on  the  top  1000  documents. 
Equation  (2)  is  used  to  calculate  the  document  vectors  as 
well. 

In  phase  2,  all  parameters  introduced  in  Section  4  are 
trained  on  last  year’s  relevance  feedback  data  set.  The  Indri 
query  string  of  our  relevance  feedback  model  is  created  as 
follows: 


#weight((l  Xfb'jQSorg  XgbQSre{^ 

where  QSorg  is  the  original  query.  QSrei  is  the  following 
query  string  created  based  on  the  relevance  model: 

#weight(P(wi\R)  wi  P(iU2\R)  w 2  •  •  •  P(wn\R)  wn ) 

6.  EXPERIMENTAL  RESULTS 

6.1  Which  documents  are  good  for  relevance 
feedback 

We  submitted  two  runs  in  phase  1:  UCSC.l  (generated 
by  TED)  and  UCSC.2  (generated  by  clustering).  Based  on 
the  evaluation  metric1  used  for  phase  1,  UCSC.l  is  ranked 
higher  than  UCSC.2,  which  means  that  the  TED  method 
performs  better  than  the  clustering  method  in  general.  Ta¬ 
ble  1  compares  the  document  sets  for  UCSC.l  and  UCSC.2. 
We  notice  that  the  TED  method  returns  more  relevant  docu¬ 
ments  (126)  than  the  clustering  method  (116).  This  is  prob¬ 
ably  why  UCSC.l  performs  better  than  UCSC.2.  However, 
UCSC.2  has  fewer  topics  with  zero  relevant  documents  (9) 
than  UCSC.l  (12).  This  is  consistent  with  what  we  ex¬ 
pected:  for  ambiguous  queries,  clustering  method  tends  to 
choose  a  diversified  set  of  documents,  so  it  is  more  likely 
that  some  relevant  documents  are  covered. 


Table  1:  Clustering  v.s.  TED 


doc  set 

method 

of  rel  docs 
returned 

#  of  topics 
with  no  rel  doc 

UCSC.2 

Clustering 

116 

9 

UCSC.l 

TED 

126 

12 

For  the  mixture  model  of  different  document  representa¬ 
tions,  let’s  look  at  the  word  ’’obama”  as  an  example.  In  this 


1See  track  overview  paper  for  details 


Figure  2:  Relevance  feedback  method.  Wi,Wj...  are  words  selected  for  query  expansion 


6.2  Performance  of  relevance  feedback  algo¬ 
rithm 

Our  submissions  in  phase  2  are  ranked  the  2nd  among  all 
participants.  This  indicates  the  effectiveness  of  our  relevance 
feedback  algorithm.  Also,  our  RF  algorithm  performs  simi¬ 
larly  well  on  each  of  the  eight  document  sets,  which  shows 
the  robustness  of  our  algorithm. 

The  topic-level  performances  of  our  RF  algorithm  are  shown 
in  table  2.  All  the  results  are  averaged  over  8  document  sets. 
The  average  retrieval  performance  (stAP)  over  all  topics  is 
improved  by  34.9%  over  the  baseline  case  where  no  RF  is 
used. 

We  also  notice  that  relevance  feedback  may  hurt  some 
topics.  Among  the  50  topics,  the  retrieval  performances  get 
worse  on  13  ones  (which  is  more  than  1/4  of  all  topics)  when 
using  relevance  feedback.  Table  3  shows  these  topics.  Fur¬ 
ther  failure  analysis  shows  that  relevance  feedback  actually 
exacerbates  the  focus  on  some  aspects  of  these  topics  and 
neglects  some  other  key  aspects. 

7.  CONCLUSIONS 

We  tried  two  methods  in  finding  good  documents  for  rel¬ 
evance  feedback.  We  found  that  the  TED  method  per¬ 
forms  better  than  the  clustering  method  mainly  because 
it  returns  more  relevant  documents.  Our  baseline  retrieval 
model  goes  beyond  the  commonly  used  ”bag-of-words”  ap¬ 
proach,  and  our  relevance  feedback  model  improves  retrieval 
performance  by  35%  over  a  baseline  without  RF. 
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Table  2:  Performances  of  relevance  feedback  on  each 
topic 


topic  id 

stAP  of 
baseline 

stAP 
with  RF 

improve 

average  # 
of  rel  docs 

overall 

0.1718 

0.2318 

34.9% 

1.97 

rf09-l 

0.4021 

0.6860 

70.6% 

1.9 

rf09-2 

0.3383 

0.2670 

-21.1% 

3.4 

rf09-3 

0.0042 

0.0124 

194.6% 

2.0 

rf09-4 

0.0465 

0.1344 

189.1% 

0.8 

rf09-5 

0.0878 

0.1049 

19.4% 

0.5 

rf09-6 

0.0872 

0.3701 

324.3% 

0.9 

rf09-7 

0.0369 

0.0262 

-29.0% 

3.5 

rf09-8 

0.0024 

0.0319 

1226.5% 

0.6 

rf09-9 

0.0365 

0.0292 

-20.1% 

3.4 

rf09-10 

0.0768 

0.3783 

392.9% 

1.6 

rf09-ll 

0.1303 

0.1752 

34.4% 

3.3 

rf09-12 

0.2155 

0.1961 

-9.0% 

2.9 

rf09-13 

0.0008 

0.0006 

-21.0% 

0.3 

rf09-14 

0.0101 

0.0265 

162.1% 

1.3 

rf09-15 

0.2886 

0.3559 

23.3% 

4.0 

rf09-16 

0.3092 

0.4999 

61.7% 

2.8 

rf09-17 

0.0972 

0.0952 

-2.0% 

1.1 

rf09-18 

0.1449 

0.2234 

54.2% 

2.0 

rf09-19 

0.0016 

0.0019 

17.7% 

0.0 

rf09-21 

0.2960 

0.3878 

31.0% 

3.6 

rf09-22 

0.4201 

0.4864 

15.8% 

4.4 

rf09-23 

0.0280 

0.0306 

9.4% 

0.9 

rf09-24 

0.0213 

0.0520 

144.6% 

2.1 

rf09-25 

0.2458 

0.4280 

74.1% 

1.4 

rf09-26 

0.1858 

0.2061 

10.9% 

2.5 

rf09-27 

0.2033 

0.2166 

6.5% 

1.8 

rf09-28 

0.5131 

0.4535 

-11.6% 

3.0 

rf09-29 

0.0483 

0.0532 

10.1% 

0.1 

rf09-30 

0.2775 

0.2478 

-10.7% 

3.4 

rf09-31 

0.1331 

0.4079 

206.6% 

4.3 

rf09-32 

0.0125 

0.0078 

-37.9% 

2.9 

rf09-33 

0.4390 

0.3959 

-9.8% 

3.4 

rf09-34 

0.0274 

0.0736 

168.5% 

0.6 

rf09-35 

0.3079 

0.3423 

11.2% 

4.0 

rf09-36 

0.0789 

0.1482 

88.0% 

1.3 

rf09-37 

0.0516 

0.0566 

9.6% 

0.3 

rf09-38 

0.1793 

0.1029 

-42.6% 

2.3 

rf09-39 

0.1436 

0.2709 

88.6% 

2.3 

rf09-40 

0.1519 

0.2286 

50.5% 

1.5 

rf09-41 

0.2137 

0.2251 

5.3% 

2.8 

rf09-42 

0.0504 

0.1396 

177.0% 

0.3 

rf09-43 

0.2324 

0.4128 

77.6% 

0.9 

rf09-44 

0.0123 

0.0592 

381.4% 

0.9 

rf09-45 

0.1854 

0.1880 

1.4% 

2.8 

rf09-46 

0.6565 

0.6366 

-3.0% 

3.4 

rf09-47 

0.4392 

0.6092 

38.7% 

2.6 

rf09-48 

0.1618 

0.2698 

66.7% 

1.3 

rf09-49 

0.2591 

0.4948 

90.9% 

1.0 

rf09-50 

0.1258 

0.1092 

-13.2% 

1.0 

Table  3:  Topic  examples  where  relevance  feedback 
(RF)  hurts 


id 

topic 

stAP 

with¬ 

out 

RF 

stAP 

improve¬ 

ment 

with  RF 

rf09-38 

dogs  for  adoption 

0.1793 

-42.6% 

rf09-32 

website  design  hosting 

0.0125 

-37.9% 

rf09-7 

air  travel  information 

0.0369 

-29.0% 

rf09-2 

french  lick  resort  and  casino 

0.3383 

-21.1% 

rf09-13 

map 

0.0008 

-21.0% 

rf09-9 

used  car  parts 

0.0365 

-20.1% 

rf09-50 

dog  heat 

0.1258 

-13.2% 

rf09-28 

inuyasha 

0.5131 

-11.6% 

rf09-30 

diabetes  education 

0.2775 

-10.7% 

rf09-33 

elliptical  trainer 

0.4390 

-9.8% 

rf09-12 

djs 

0.2155 

-9.0% 

rf09-46 

alexian  brothers  hospital 

0.6565 

-3.0% 

rf09-17 

poker  tournaments 

0.0972 

-2.0% 

